1. Field of the Invention
The present invention relates to an image recognition apparatus for recognizing objects using the image processing technology and an image processing apparatus for recognizing road surface indications in an image of a road surface.
2. Description of the Related Art
There are vehicles such as automobiles that are equipped with a drive assisting apparatus or an automatic driving apparatus. In many cases, drive assisting apparatuses and automatic driving apparatuses have an image recognition apparatus for recognizing objects ahead of the vehicle. Where an image recognition apparatus performs double-eye ranging using the image processing technology, the image recognition apparatus is equipped with a right camera and a left camera. Provided so that a parallax occurs between images taken, the right camera and the left camera shoot objects in the space ahead of the vehicle. In the image recognition apparatuses, when the left camera and right camera produce a left input image and a right input image shown in FIGS. 15A and 15B, respectively, the position and strength of each longitudinal edge in the right input image and those of each longitudinal edge in the left input image are determined first. As a result, a left edge image shown in FIG. 15C and a right edge image shown in FIG. 15D are obtained. Then, based on the determined position and strength of each longitudinal edge, the left edge image and the right edge image are matched with each other. Parallaxes between images in the left input image and the right input image are obtained as a matching result. The distances between the objects and the vehicle are determined.
In the above conventional double-eye image recognition apparatus having two cameras, longitudinal edges in a left input image and those in a right input image are matched with each other to calculate parallaxes. The matching between the edges often produces errors. Furthermore, since only the longitudinal edges are used, it is difficult to determine a corresponding relationship between lines in a left input image and a right input image that extend obliquely with respect to the longitudinal direction.
Where a left input image and a right input image are obtained by shooting the space ahead of a vehicle that is equipped with the image recognition apparatus, if input images contain a motorcycle or another vehicle that is cutting in ahead, many of the edges of such a vehicle or motorcycle in the input images are oblique lines. Since the conventional double-eye image recognition apparatus has difficulty recognizing a corresponding relationship between oblique lines, it is difficult for the image recognition apparatus to recognize a motorcycle or vehicle that is cutting in ahead of the vehicle that is equipped with the image recognition apparatus.
Where a left input image and a right input image are obtained by shooting the space ahead of a vehicle that is running on a road, the left input image and the right input image contain the surface of the road and road surface indications that are drawn on the road. If there is no object ahead of the left and right cameras, edges of road surface indications are determined in detecting edges in left and right images. The edges of the road surface indications may be recognized erroneously as edges of objects, whereupon object recognition and double-eye ranging are performed. That is, in the conventional double-eye image recognition apparatus, errors frequently occur in object recognition and double-eye ranging when there is no object ahead of the left and right cameras.
An object of the present invention is to provide an image recognition apparatus capable of stably recognizing an object with a less number of errors as well as recognizing an object having many oblique edges. Another object of the invention is to provide an image processing apparatus capable of reliably recognizing road surface indications in a displayed image of a road surface.
The invention provides an image recognition apparatus comprising:
first and second cameras provided so as to produce a parallax, for shooting in respective visual fields;
edge detecting means for detecting edges in a first input image that is supplied from the first camera;
matching means for extracting, as a reference pattern, a portion in the first input image that includes at least part of a detected edge and for conducting matching of the reference pattern with a second input image that is supplied from the second camera using a pattern matching method; and
object recognizing means for recognizing an object in the visual fields of the first and second cameras, based on a matching result.
According to the invention, the image recognition apparatus recognizes an object in the visual fields of the first and second cameras, based on a parallax that is indicated by a result of matching between a reference pattern and a second input image. To calculate the parallax of the object, the reference pattern including at least part of an edge of the object in the first input image that is supplied from the first camera is matched with the second input image that is supplied from the second camera by using the pattern matching method. This reduces the possibility of occurrence of errors in the object recognition based on a matching result and enables the object recognition to be performed stably.
The image recognition apparatus of the invention is characterized by further comprising road surface indication recognizing means for recognizing road surface indications contained in at least one input image of the first input image and the second input image, and characterized in that the edge detecting means detects remaining edges other than edges of the road surface indications among all the edges in the first input image, from the first input image, based on a road surface indication recognition result.
According to the invention, road surface indications contained in at least one of the first input image and the second input image are recognized and only remaining edges other than the edges of the road surface indications among all the edges in the first input image are detected. The matching means conducts matching between a reference pattern which is a portion in the first input image that includes at least part of a remaining edge, and the second input image. This prevents the object recognizing means from recognizing road surface indications in recognizing objects in the visual fields. Since the edges of road surface indications in the first input image are removed from the edge detection result of the first input image, the recognition accuracy of objects other than road surface indications in the visual fields is increased.
The image recognition apparatus of the invention is characterized in that the road surface indication recognizing means recognizes only the road surface indications in the first input image.
According to the invention, only the road surface indications in the first input image are recognized prior to the edge detection. Therefore, the image recognition apparatus of the invention can reduce the load of processing of the road surface indication recognizing means as compared with an image recognition apparatus in which road surface indications in both the first input image and the second input image are recognized.
The image recognition apparatus of the invention is characterized in that:
the first camera and the second camera are disposed on a mounting axial line that is parallel with a road surface in the visual field so as to be separated from each other by a predetermined distance,
the road surface indication recognizing means:
(1) divides the at least one input image into a plurality of band-like regions extending in a reference direction corresponding to a direction parallel with the mounting axial line in the at least one input image,
(2) on the band-like region basis, carries out a search of density variation pattern in the reference direction to determine as a candidate portion for a road surface indication, a portion having one end whose density increases according to a predetermined reference increase pattern and another end whose density decreases according to a predetermined reference decrease pattern, and
(3) compares a width of the candidate portion for a road surface indication with a predetermined reference width, and when the width of the candidate portion for a road surface indication is smaller than or equal to the reference width, recognizes that the candidate portion in the at least one input image is part of a road surface indication, and
the reference width is predetermined based on the width of a road surface indication that is closest to the first camera and the second camera in a displayed image of a road surface.
According to the invention, the road surface indication recognizing means recognizes that a portion in the at least one input image of the first input image and the second input image is part of a road surface indication, the portion being such that the density increases and decreases according to the predetermined reference increase pattern and reference decrease pattern in the reference direction corresponding to the direction parallel with a road surface and its width in the reference direction is smaller than or equal to the reference width. This allows the road surface indication recognizing means to easily recognize a portion in the input image that is part of a road surface indication, based on only a density variation pattern in the reference direction.
The image recognition apparatus of the invention is characterized in that the reference increase pattern is a density variation pattern of from a density corresponding to black to a density corresponding to white and the reference decrease pattern is a density variation pattern of from the density corresponding to white to the density corresponding to black.
According to the invention, the road surface indication recognizing means regards, as a candidate portion for a road surface indication, a portion in the at least one input image in which in the reference direction the density varies directly from the density corresponding to black to the density corresponding to white, remains the density corresponding to white over a predetermined width or more, and varies directly from the density corresponding to white to the density corresponding to black. This allows the road surface indication recognizing means to easily recognize a portion in the input image that is part of the road surface indication, based on the density variation pattern in the reference direction.
The image recognition apparatus of the invention is characterized in that the reference increase pattern is a density variation pattern of from a density corresponding to black to a density corresponding to white via a density corresponding to gray, and
the reference decrease pattern is a density variation pattern of from the density corresponding to white to the density corresponding to black via the density corresponding to gray.
According to the image recognition apparatus, the road surface indication recognizing means regards, as a candidate portion for a road surface indication, a portion in the at least one input image which has in the reference direction a density variation in which the density corresponding to black is varied via the density corresponding to gray to the density corresponding to white, which is maintained over a predetermined width or more, and varied to the density corresponding to black via the density corresponding to gray. This allows the road surface indication recognizing means to easily recognize a portion in the input image that is part of a road surface indication, based on a density variation pattern in the reference direction even if the road surface indication in the visual fields of the first camera and the second camera is blurred or stained.
The image recognition apparatus of the invention is characterized in that the reference increase pattern is a density variation pattern of from a density corresponding to black via a density corresponding to gray to a density corresponding to white, and
the reference decrease pattern is a density variation pattern of from a density corresponding to white to a density corresponding to black.
According to the image recognition apparatus, the road surface indication recognizing means regards, as a candidate portion for a road surface indication, a portion in the at least one input image which has in the reference direction a density variation in which the density corresponding to black is varied via the density corresponding to gray to the density corresponding to white, which is maintained over a predetermined width or more, and directly varied to the density corresponding to black. This allows the road surface indication recognizing means to easily recognize a portion in the input image that is part of a road surface indication, based on a density variation pattern in the reference direction even if one side end of the road surface indication in the visual fields of the first camera and the second camera is blurred or stained.
The image recognition apparatus of the invention is characterized in that the reference increase pattern is a density variation pattern of from a density corresponding to black to a density corresponding to white, and
the reference decrease pattern is a density variation pattern of from the density corresponding to white to the density corresponding to black via the density corresponding to gray.
According to the image recognition apparatus, the road surface indication recognizing means regards, as a candidate portion for a road surface indication, a portion in the at least one input image which has in the reference direction a density variation in which the density corresponding to black is directly varied to the density corresponding to white, which is maintained over a predetermined width or more, and varied to the density corresponding to black via the density corresponding to gray. This allows the road surface indication recognizing means to easily recognize a portion in the input image that is part of a road surface indication, based on a density variation pattern in the reference direction even if one side end of the road surface indication in the visual fields of the first camera and the second camera is blurred or stained.
The image recognition apparatus of the invention is characterized in that the road surface indication recognizing means further determines a number of candidate portions that have a width smaller than or equal to the reference width and that are arranged consecutively in a direction approximately perpendicular to the reference direction, and
when the number of the candidate portions is greater than or equal to a predetermined reference number, recognizes that the candidate portions include parts of an image of a road surface indication in the at least one input image.
According to the invention, the road surface indication recognizing means recognizes that among all the candidate portions for road surface indications in the at least one input image candidate portions whose widths in the reference direction are smaller than or equal to the reference width and that are arranged consecutively in a direction approximately perpendicular to the reference direction in a number greater than or equal to the reference number are a road surface indication. This enables the road surface indication recognizing means to increase the recognition accuracy of road surface indications in the input image.
The image recognition apparatus of the invention is characterized in that the road surface indication recognizing means judges whether an approximately band-like region that consists of the candidate portions that have a width smaller than or equal to the reference width and that are arranged consecutively in the direction approximately perpendicular to the reference direction in the predetermined number or more has an inclination that is within an allowable range that is predetermined based on an inclination of road surface indications in a displayed image of a road surface, and
only when the inclination of the approximately band-like region is within the allowable range, recognizes that the candidate portions constituting the approximately band-like region is a road surface indication.
According to the invention, the road surface indication recognizing means recognizes that among all the candidate portions for road surface indications in the at least one input image candidate portions whose widths in the reference direction are smaller than or equal to the reference width and that are arranged consecutively in a direction approximately perpendicular to the reference direction in a number greater than or equal to the reference number are a load surface indication only if the inclination of an approximately band-like region consisting of those candidate portions is within the allowable range. This enables the road surface indication recognizing means to further increase the recognition accuracy of road surface indications in the input image.
The image recognition apparatus of the invention is characterized in that the road surface indication recognizing means searches an entire area of the at least one input image for candidate portions for a road surface indication.
According to the invention, the road surface indication recognizing means searches the entire area of the received at least one input image for candidate portions for a road surface indication. This allows the road surface indication recognizing means to recognize road surface indications in the input image reliably, that is, without failing to recognize part of those.
The image recognition apparatus is characterized in that the road surface indication recognizing means searches, for candidate portions for a road surface indication, a prediction region of the at least one input image where a road surface indication is expected to exist.
According to the invention, the road surface indication recognizing means searches, for candidate portions for a road surface indication, only a prediction region of the received at least one input image where a road surface indication is expected to exist. This reduces the processing load of the road surface indication recognizing means and shortens the time required for the recognition of road surface indications.
The image recognition apparatus of the invention is characterized in that the edge detecting means:
(1) judges, for a pixel basis, whether pixels in the first input image are located outside road surface indication portions therein, based on a road surface indication recognition result; and
(2) performs, only to each of pixels in the first input image that are located outside the road surface indication portions therein, a process for judging whether the pixel constitutes an edge in the first input image.
According to the invention, the edge detecting means performs calculation for edge detection on only pixels in the first input image that are located outside road surface indication portions. This enables the edge detecting means to reduce the time required for detection of edges in the first input image other than the edges of road surface indications and to decrease the related processing load.
The image recognition apparatus of the invention is characterized in that the first input image and the second input image each consist of a plurality of pixels, and
the matching means conducts matching between the reference pattern and the second input image, using color densities of the respective pixels as indices.
According to the invention, the matching means uses the color densities of the respective pixels as indices in matching the reference pattern with the second input image using the pattern matching method. This allows the matching means to easily conduct matching between the reference pattern and the second input image.
The image recognition apparatus of the invention is characterized in that the matching means corrects, prior to the pattern matching, the densities of the respective pixels of at least one input image of the first input image and the second input image using an offset value commensurate with an individual, color-related difference between the first camera and the second camera.
According to the invention, the matching means performs setting of a reference pattern and matching calculation after correcting at least one input image of the first input image and the second input image in accordance with an individual, color-related difference between the cameras. This enables the matching means to increase the matching accuracy.
The image recognition apparatus of the invention is characterized in that the matching means learns the individual, color-related difference between the first camera and the second camera, based on the first input image and the second input image.
According to the invention, the matching means learns the individual, color-related difference between the cameras and uses the learned individual difference for the correction. This enables the matching means to further increase the matching accuracy when the left camera and the right camera have an individual difference.
The image recognition apparatus of the invention is characterized in that the first input image and the second input image each consist of a plurality of pixels, and
the matching means conducts matching between the reference pattern and the second input image, using color density variation amounts of the pixels as indices.
According to the invention, this image recognition apparatus, the matching means uses color density variation amounts of the pixels as indices in matching the reference pattern with the second input image using the pattern matching method. This allows the matching means to easily conduct matching between the reference pattern and the second input image. For example, the color density variation amount of a pixel is the difference between the color densities of two pixels adjacent to each other. Using color density variation amounts of the pixels as indices enables the matching means to increase the matching accuracy.
The image recognition apparatus of the invention is characterized in that the matching means calculates a density difference between minimum and maximum densities of the pixels of the reference pattern and only when the calculated density difference is greater than or equal to a predetermined reference density difference, uses the reference pattern for the pattern matching,
the predetermined reference density difference being predetermined based on a minimum density difference of a reference pattern that enables effective matching.
According to the invention, in the reference pattern that is used for the pattern matching, the density difference between the maximum density and the minimum density of all the pixels of the reference pattern is always greater than or equal to the reference density difference. This makes a result of the matching between the reference pattern and the second input image always effective. Since reference patterns whose density differences are smaller than the reference density difference among the extracted reference patterns are not used, no time is consumed to perform unnecessary pattern matching operations. Therefore, the time necessary for the matching of all the reference patterns is shortened.
The image recognition apparatus of the invention is characterized in that the first camera and the second camera are disposed on a predetermined mounting axial line so as to be separated from each other by a predetermined distance,
in performing the pattern matching the matching means sets a search region as a subject of a process of the pattern matching in the second input image and compares, with a density distribution of the reference pattern, a density distribution of a portion in the second input image which portion includes at least one pixel of the search region and is congruous with the reference pattern,
a width of the search region in a direction that is perpendicular to a reference direction corresponding to a direction parallel with the mounting axial line in the second input image is greater than or equal to three pixels, and
a coordinate in the perpendicular direction of a center of the search region in the second input image is the same as a coordinate in the perpendicular direction of a center of the reference pattern in the first input image.
According to the invention, the matching means sets a search region in the second input image. The width of the search region in a direction that is perpendicular to a reference direction that corresponds to a direction parallel with the mounting axial line of the right and left cameras is greater than or equal to a width whose ends in the perpendicular direction are deviated from the pixel of the second input image that corresponds to the central pixel of the reference pattern by several pixels (e.g., one pixel). Where a search region is set, this makes it possible to produce a result of matching between the reference pattern and the second input image without causing an error even if the positional relationship between the optical axes of the first camera and the second camera is deviated from the design positional relationship.
The image recognition apparatus of the invention is characterized in that the first camera and the second camera are disposed on a predetermined mounting axial line so as to be separated from each other by a predetermined distance,
in performing the pattern matching the matching means sets a search region as a subject of a process of the pattern matching in the second input image and compares, with a density distribution of the reference pattern, a density distribution of a portion in the second input image which portion includes at least one pixel of the search region and is congruous with the reference pattern,
a width of the search region in a reference direction corresponding to a direction parallel with the mounting axial line in the second input image is a width commensurate with an upper limit parallax that is determined based on an arrangement of the first camera and the second camera, and
a coordinate in the reference direction of a center of the search region in the second input image is deviated from a coordinate in the reference direction of a center of the reference pattern in the first input image in the same direction as a deviation corresponding to a parallax of the second input image with respect to the first input image.
According to the invention, the matching means sets a search region in the second input image. The search region is elongated with respect to the pixel in the second input image that has the same coordinate as the central pixel of the reference pattern in the first input image only in the direction of a deviation of the second input image with respect to the first input image that corresponds to a parallax. The width of the search region in the reference direction is commensurate with an upper limit parallax in the first input image and the second input image. This makes it possible to minimize the possibility of occurrence of a matching error and shorten the time necessary for the pattern matching.
The image recognition apparatus of the invention is characterized in that in performing the pattern matching the matching means:
(1) selects, one by one, a plurality of detected edges in the first input image in order from an edge having highest edge intensity;
(2) every time an edge is selected, sets, in the first input image, a reference pattern including at least part of the selected edge;
(3) sets, in the second input image, a search region as a processing subject of the pattern matching, based on the position of the set reference pattern in the first input image and a matching result of a reference pattern that has already been processed and the second input image; and
(4) compares a density distribution of the set reference pattern and a density distribution of a portion in the second input image that includes at least one pixel of the search region and is congruous with the set reference pattern.
According to the invention, the matching means perform the setting of reference patterns and search regions and the pattern matching calculation on a plurality of edges in descending order of edge intensity. Every time an edge is selected, a search region corresponding to the reference pattern is set in the second input image, based on not only the position of the reference pattern in the first input image but also a result of pattern matching that has already been performed. Since each search region can be made narrower than in a case where each search region is set based on only the position of the corresponding reference pattern in the first input image, the time necessary for the pattern matching can further be shortened.
The invention also provides an image processing apparatus comprising road surface image storing means for storing a displayed image of a road surface; and road surface indication recognizing means for recognizing a displayed portion of a road surface indication in the road surface image, wherein the road surface indication recognizing means
(1) divides the road surface image into a plurality of band-like regions extending in a reference direction corresponding to a direction parallel with the road in the image;
(2) on the band-like region basis, carries out a search of density variation pattern in the reference direction to determine as a candidate portion for a road surface indication, a portion having one end whose density increases according to a predetermined reference increase pattern and another end whose density decreases according to a predetermined reference decrease pattern, and
(3) compares a width of the searched candidate portion for a road surface indication with a predetermined reference width, and when the width of the searched candidate portion for a road surface indication is smaller than or equal to the reference width, recognizes that the searched candidate portion is part of a road surface indication, and
the reference width is predetermined based on the width of a road surface indication that is closest to a camera that shot a road surface in a displayed image of the road surface.
According to the invention, the road surface indication recognizing means recognizes that a portion in an image of a road surface is part of a road surface indication, the portion being such that the density increases and decreases according to the predetermined reference increase pattern and reference decrease pattern in the reference direction that corresponds to the direction parallel with the road surface and its width in the reference direction is smaller than or equal to the reference width. This allows the road surface indication recognizing means to easily recognize a road surface indication in a received image, based on only a density variation pattern in the reference direction.